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ABSTRACT 



In May 1972, 2,900 copies of a survey were distributed 
eliciting opinions about the meaning of some qualifying adjectives and 
phrases commonly used in "verbal communications." These same adjective 
phrases are commonly used in survey research. They include phrases such as "a 
few," "a majority," "nearly all," and other quantifiers. The 1972 survey had 
a 40% response rate, indicative of interest in the study. This study updates 
that 1972 study, reporting findings from different groups of subjects. Nine 
groups (graduate students in several disciplines, including education, 
undergraduates, and engineers and administrators) ranging in size from 3 to 
20 members were surveyed. Results made it clear that individuals within 
different groups give different interpretations to quantifying phrases, and 
that differences also exist among groups. It is suggested that there is a 
need to study item reliability in surveys in more detail if words and 
phrases, such as the 22 identified in this study, are to be included in 
response options, narrative instructions in a survey, or reporting on the 
survey findings. Appendix A is the report from the 1972 survey, and Appendix 
B contains a copy of the original survey form. (Contains five tables, eight 
figures, and three references.) (SLD) 



***************************************************************************** 

♦ Reproductions supplied by EDRS are the best that can be made 

♦ from the original document. 



TM028415 ed 419 839 



5 






WORDS AND THEIR VALUE 
TO THE 

SURVEY RESEARCHER 



Chester H. McCall 

Graduate School of Education and Psychology 
Pepperdine University 
Culver City, CA 90230 

Lauren E. Walters 
Teacher 

Los Angeles Unified School District 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

Z ATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

s document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

C,. //lcra.11 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

H 



American Eklucational Research Association 
1998 Annual Meeting 
San Diego, CA 

April 13-17, 1998 




AERAU998.doc 



CMCCALL@PEPPERDINE.EDU 



4/3/98 



2 



WORDS AND THEIR VALUE TO 
THE SURVEY RESEARCHER 

DESCRIPTORS: 

Survey research, research methodology, validity/reliability, instruments 
Background 



In May of 1972, 2,900 copies of a survey were distributed eliciting opinions regarding the meaning of 
some qualifying adjectives/phrases commonly used in “verbal communications.” These same 
adjectives/phrases are commonly used in survey research. John Hoyt (1972) in his report emphasizes the 
problems which arise when using some of these words and/or phrases, particularly when decisions might 
be made based upon an interpretation of what these qualifying adjectives/phrases might mean. In the 1972 
paper, the author does not identify how the 2,900 initial sample was selected. He indicates that within 
three weeks a 40% response was obtained, “demonstrating that there was ‘rather considerable’ interest in 
the subject.” Stanley Payne (1951) in his provocative book points out how important it is in developing an 
instrument with questions that the meanings of the allowable responses be as clear as possible. Paul 
Scipione (1995) in his timely article includes the following introductory comments: 

“Those of us who design and direct research studies spend so much time with numbers that we 
often forget that words also represent values (in the minds of the writer as well as the readers; 
authors comments). Yet both the questionnaires we use to elicit information from respondents 
and the reports we use to communicate research findings and recommendations to clients depend 
significantly on language.” 

He goes on to make several additional points which, independently, have been raised after more than 15 
years of survey research involvement by the author of this paper: 

• We speak of a substantial change rather than indicating the numerical amount of the change; 

• We convert descriptive words to numerical values and develop descriptive statistics, such as 
averages, to discuss the findings; and, 

• We make considerable use of descriptive words and phrases, such as a lot, many, or 
frequently, in providing a report on the findings of a survey. 

The descriptive results of the 1972 study appear as Appendix A to this paper. A copy of the initial survey 
form appears as Appendix B. It is important to note that these two appendices are exactly as reported by 
John Hoyt in 1972. 

Objectives/Purposes 

Believing strongly that the 1972 results have probably not changed much and, because of the author’s 
interest in using surveys to collect useful information in educational research and for making management 
decisions both within education and elsewhere, the author received release time from his educational 
institution to collect additional data using the same survey and the same set of instructions. Had the author 
been aware of the Scipione (1995) study at the time of his research, the items used on the instrument might 
have been slightly modified; although, from a research perspective, a replication of the exact instrument 
and instructions (Hoyt, 1972) seemed most appropriate. 

The purpose of this study was to update the 1972 study and to report findings from groups of different 
subjects. It was hypothesized that individuals who might become subjects for surveys would continue to 
attach different meanings or interpretations to the words and phrases in the original study, creating a need 
for an awareness on the part of those using the survey results about the possible differences in 
interpretation of the words and phrases, not only as instruments are developed but also as the results are 
reported. 
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Perspectives/Theoretical Framework 



Of considerable concern in survey research is assessing the reliability and validity of an instrument and 
also items on the instrument. In the context of this paper, reliability of an item is the extent to which the 
item means the same thing to all respondents at a given time and the same thing to an individual respondent 
at different points in time. It is generally agreed that survey results may be, essentially, worthless if the 
individual items on the instrument are unreliable and invalid. This paper addresses the reliability issue in 
terms of the way in which words or phrases used in assessing opinions or perceptions may be interpreted 
by the respondent and, hence, interpreted and reported by the researcher. The instrument used in the 1972 
study and again in this study contains 22 items. 

Methods. Techniques. Modes of Inquiry 

Unlike the 1972 study in which the sampling procedure was not clearly identified, the intent of this 
replication was to utilize individuals from nine different groups to address two questions: 

• Do those responding today produce similar results (in terms of averages, medians, modes, 
ranges, and distributions) to those obtained in 1972? 

• Is there a difference in the response patterns among nine different groups? 

The instrument was administered in a group setting by the author and by three other faculty members at a 
single institution. Because of the ambiguity of the 1972 instrument (Appendix B), faculty explained what 
was expected in assigning numerical values to the verbal descriptions. Data were coded, entered onto an 
EXCEL spread sheet and imported into NCSS 97© 6.0 for Windows, a statistical analysis software 
package. 

Data Sources or Evidence 

The nine groups from which data were collected were: 

1 . A research class at the doctoral level, fall 1995. 

2. A special class of doctoral students studying “organization change,” fall 1995. 

3. A research class at the doctoral level, winter 1996. 

4. Undergraduate students in a communications major, fall 1995. 

5. An education faculty associated with graduate programs, winter 1996. 

6. A third research class at the doctoral level, winter 1996. 

7. A masters class majoring in educational administration, winter 1996. 

8. A masters class in research with students primarily in teacher training education, spring 1996. 

9. A group of engineers and administrators in the management construction business during a 
seminar on survey methods, summer 1996. 

The nine groups above ranged in size from nine to twenty three individuals. After data were coded and 
summary statistics obtained, individual responses were reviewed for logic and consistency. In three cases 
the results were discarded because the respondent apparently did not understand the directions. 
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Results and/or Conclusions/Point of View 



Summary statistics for the 1972 data appear as Appendix C. Appendix D (Tables 1 through 5) indicate for 
each of the nine groups and each of the 22 quantifying phrases, the following statistics: 

• Arithmetic mean (average) 

• Median 

• Mode 

• The range of reported values 

• The minimum and maximum reported values 

It is interesting to note, for selected quantifying phrases and the arithmetic mean, median, and minimum 
and maximum values, comparisons among the nine groups and the Hoyt (1972) study: 



ARITHMETIC MEAN 




Hoyt 


1 


2 


3 


4 


5 


6 


8" 


9 


A few 


8.31 


7.4 


17.1 


9.8 


3.9 


17.4 


8.2 


10.1 


6.4 


Many 


31-40' 


54.3 


45.7 


51.7 


30.4 


50.3 


49.9 


45.8 


44.9 


A majority 


51-55 


59.8 


56.7 


56.9 


57.9 


56.6 


66.4 


68.7 


56.8 


Nearly all 


86-90 


95.4 


87.8 


94.7 


85.8 


84 


95.5 


88.1 


85.6 


MODE" 




A few 


10 


3 


3 


3 


3 




5 


3 


3 


Many 


40 


50 




80 




60 




40 


25 


A majority 


51 


51 


51 


51 


51 


51 


51 


51 


51 


Nearly all 


90 


98 


90 




95 


95 




90 




MAXIMUM AND MINIMUM 




A few 


N/A* 


0-25 


3-80 


3-50 


2-12 


3-80 


2-25 


3-60 


0-30 


Many 


N/A 


20-100 


[75-80 


10-100 


4-80 


20-90 


20-99 


6-90 


5-100 


A majority 


N/A 


51-90 


51-90 


10-90 


10-99 


51-90 


51-100 


40-98 


25-100 


Nearly all 


N/A 


90-99 


20-98 


85-99 


20-99 


0-99 


89-99 


5-99 


25-99 























2 - Data from Group 7 in Appendix D appear to be entered in error and have not been included in this table. 

3 - Due to small samples the mode did not exist in some of the groups. 

4 - Hoyt did not report ranges. 



Detail for the other phrases and groups are reported in Appendix D. 

To give a flavor of the specific relative frequency of responses. Appendix E contains histograms for the 
following phrases: 



• A couple 

• A few 

• A majority 

• A substantial majority 

• Numerous 

• Lots 

• A large proportion of 

• Virtually all 




AERANPAPER-98 . DOC 



PAGE 



3 



5 



4/6/98 



Even if we assume that some of the respondents may not have understood the directions or were “joking 
around, the results still are unnerving and suggest the need for great caution in using quantifying phrases 
in both the body of surveys and reporting the findings of surveys. 

It is quite clear that individuals within groups give different interpretations to quantifying phrases and 
among groups differences also exist. It is suggested, although not necessarily supported by the data in this 
study, that different professions might exhibit differences in not only location statistics but also ranges. 

Educational or Scientific Importance of the Study 

It is clear to the author that the diverse opinions with respect to the meaning of many of the “quantifying” 
words and phrases are pause for reflection on how we are analyzing and reporting survey results. It is 
suggested that there is a need to pursue further the issue of item reliability if words and phrases, such as the 
22 identified in this study, are to be included in response options, in narrative instructions in a survey, or in 
reporting on the survey findings. 

REFERENCES 
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NOTE: Scipione s paper includes 17 excellent references related to both verbal and visual presentations of 
items associated with questionnaires and their analyses. 
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APPENDIX A 



RESULTS OF THE SURVEY 



General 



The first, and probably strongest, point to be made in looking at the survey 
results is that, despite the large number of valid responses (1180) and the 
rate of return (40^) to the survey, no claim is made for the precision 
of the results. Indeed, the most valid single conclusion that can be drawn 
is that the quantifying adjectives used in the survey represent to the sample 
of respondents surveyed a broad range of numbers. It is also clear that the 
context in which the adjective is used may be as relevant to the quantity implied 
as is the quantifying phrase itself. In addition, the results suggest to all who 
verbalize quantitative data that they ought to be as precise as possible and, to 
those of us who listen to verbalized data, to insist that the speaker inject 
precision into his verbalizations. 

The meanings of the quantifying adjectives are summarized briefly in figure 1 
and graphically in figures 2 through 25. In addition, for those who are con- 
cerned with additional statistical analysis a printout of the 1180 valid responses 
for each of the quantifying adjectives is available from MAlPS . 

The graphical representations, in addition to presenting the distribution of the 
responses, also show three statistics for each response. First, the "mode" 
which is simply the discrete number that appeared most often for each adjective. 
Second, the "median" which is the number in the series of responses that represents 
a point such that half of the numerical responses fall on one side of it and half 
bhe other side. Third, the '*mean" which is the sum of all of the responses 
divided by the number of responses. The mean is also frequently referred to as 
the average ' response. To illustrate, if a group of 9 numerical responses were; 
1»1»2,2,3,4,4,4,6 — then the mode would be 4 (the most frequent response); the 
median would be 3 (there are as many responses below 3 as there are above 3); 
and the mean would be 3 (the sum of all the responses, 29, divided by the number 
of responses, 9). 

Another general observation about the responses is that, with the exception of 
quantifying adjectives which represent numbers less than 10 or numbers greater 
than 90, the most frequent responses fell on numbers divisible by 5. There are 
two other exceptions to this observation; the phrase 'a minority' and the phrase 
*a majority' are subject to rather precise statistical definition and the responses 
were, as one would expect, most frequently given as these two single ntimbers . 

For responses where the quantifying adjective has a mode of 90 or above there 
were relatively frequent usages of the numbers 95, 98, and 99 and a relatively • 
infrequent choice of the other discrete numbers between 90 and 100. 
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appendix b 



Out of a total of 100 items (votes, apples, dollars, opinions, or what have 
you) place a number in the right hand column that represents your under- 
standing of each of the following descriptors. (Please use only one whole 
number as your "best" answer; no decimals, fractions, or ranges). Remember, 
enter the whole number out of 100(%) for each description. 



—. scrip tor Number it Represents 

"A few" 

"Damn few" 

"A couple" ' ~ 

"Lots" ~ 

"A majority" 

"Several" 

"Many" 

"Most" 

"Almost all" 

"Hardly any" 

"A substantial majority" 

"A significant number of . . 

"Virtually all" 

"Almost none" 

"A small number of" 

"Numerous" 

"Nearly all" 

"A minority of" 

"A consensus of" 

"Not very man of" 

"A large proportion of" 

"A clear mandate" (i.e. votes out 

of the 100) 




SUMMARY OF SURVEY RESPONSKS APPENDIX C 



Quantifying Phrase 




Response 


Statistic 








Mode 

(Most Frequent) 


Median* 

(Mid-Point) 


Mean* 

(Average) 


Almost None 




1 


2.19 


2.84 


A Couple 




2 


2.04 


2.21 


Damn Few 




3 


3.17 


3.81 


Several 




3 


7.47 


11.94 


Hardly Any 




5 


4.52 


4.84 


Few 




10 


7.08 


8.31 


Small Number of 




10 


8.16 


9.02 


Not Very Many 




10 


6-10 


6-10 


Lots 




40 


41 - 50 


41 - 50 


Many 


i 


40 


31 - 40 


31 ,- 40 


Significant Number of 




40 


31 - 40 


31 - 40 


Considerable Number of 




40 


31 - 40 


31 - 40 


Numerous 




40 


31 - 40 


31 - 40 


Minority 




49 


21 - 30 


21 - 30 


Majority 




51 


51 - 55 


51 - 55 


Consensus 




60 


51 - 60 


51 - 60 


Substantial Majority 




75 


66 - 70 


66 - 70 


Large Proportion 




75 


61 - 70 


51 - 60 


Clear Mandate 




75 


61 - 70 


61 - 70 


Most 




90 


71 - 80 


61 - 70 


Nearly All 




90 


86 - 90 


86 - 90 


Almost All / 




95 


86 - 90 


86 - 90 


Virtually All 




95 


95.25 


94.04 



* In order to make the statistics of general usefulness medians and means are 
reported as ranges in those cases when appropriate. 
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TABLE 1. Arithmetic Mean for Nine Groups on Twenty Two Items 



appendix D 







O 

a 

•D 

2 

0 


CN 

lO 





CC) 


iv! 


o 

GO 


56.8 


o 

CO 


o 


o 

iO 


CN 

00 

00 


CN 

d 


68.7 


50.2 


<> 

00 


CO 

c6 


1 9 U 


39.3 


o 

lO 

OO 


30.4 


1 60Z 


CO 


56.9 


67.7 


MEAN 
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1 29.3 


1 10.1 


3.2 


1 6.4 


o' 


68.7 


22.9 


45.8 


81.7 
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to 

00 


1 9'IZ 


58.3 
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6.8 


00 


41.6 1 


88.1 1 


26.6 1 


68.3 1 


12.5 


66.2 


59.5 


a 

D 

2 

0 


29.3 


1 10.1 


3.2 


6.4 


1 46.1 


68.7 


22.9 


45.8 


81.7 


86.3 


to 

00 


71.5 


58.3 


89.4 1 


6.9 


00 


41.6 


55.1 


26.6 1 


68.3 


12.5 


66.2 


59.5 


1 Group 6 


40.3 


CN 

oci 


4.7 


1 10.9 


1 62.6 


66.4 


16.5 


49.9 


81.2 


85.5 


8.6 


74.4 


54.2 


896 


00 
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39.4 


95.5 


49.1 1 


81.5 


15.5 


68.4 


82 


1 Groups 


53 


1 17.4 


4.3 


901 


46.6 


56.6 


21.1 


50.3 


78 


92.4 


2.8 


61.5 


54.3 


85.7 


2.2 


7.7 


39.5 


84 


41.6 1 


84.5 


8.3 1 


70.5 
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1 3.9 


3.9 
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34.9 


57.9 


13.1 


30.4 
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00 


6.2 


72.8 1 


49.3 


90.7 


2.8 


9'6 


42.3 1 


85.8 


21.1 1 


59.4 1 


10.5 1 


70.5 


6' 18 


< 


1 Group 3 




1 9.8 


4.5 


2.9 


58.7 


56.9 


LO 


51.7 


85.4 


94.4 




77.2 1 


909 


95 1 


69 


-I 


51.2 


94.7 


33.9 1 


73.1 1 
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71.5 


77.3 
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637 


56.7 


17.3 


45.7 


76.7 


93.3 


14.1 


69.5 


46.7 ; 1 


94.8 : 1 


10.4 


16.6 


50.7 1 


87.8 1 


42.3 1 


74.7 1 


to 


70 1 


68.7 




[ Group 1 
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1 60.2 
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23.3 


54.3 


00 


95.8 


4.6 


76.5 


57.4 


87.9 


CN 


6.4 


44.6 1 


95.4 1 


24.4 1 


73.5 1 


12.9 1 


74 


84.4 




Quantifying Phrase 


Age 


A Few 


A Damn Few 


A Couple 


[Lots 


A Majority 


Several 


Many 


[Most 


< 

CO 

o 

E 

< 


Hardly Any 


|A Substantial Majori] 


1 A Significant Numb( 


[Virtually All 


Almost None 


A Small Number of 


[Numerous 


< 

b 

0) 

z 


A Minority of [ 


[a Consensus of | 


Not Very Many of 


A Large Proportion \ 


A Clear Mandate 
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TABLE 2. Median for Nine Groups on Twenty Two Items 
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Quantifying Phrase 
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A Few 


A Damn Few 


A Couple 


Lots 


A Majority 


Several 


[Many 


Most 


Almost All 


Hardly Any 


A Substantial Majorn 


A Significant Numbd 


Virtually All 


Almost None 


A Small Number of | 


Numerous 


Nearly All 


A Minority of 


A Consensus of 


Not Very Many of 


A Large Proportion j 


1 A Clear Mandate | 




TABLE 3. Mode for Nine Groups on Twenty Two Items 
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